A Cross-Jobs-Cross-Phases Map-Reduce Scheduling Algorithm in Heterogeneous Cloud

To fast process the large-scale data, map-reduce cloud is viewed as a very reasonable and effective platform. According to the new scheduling challenges in map-reduce cloud, a cross-jobs-cross-phases (CJCP) map-reduce scheduling algorithm is proposed in this paper. CJCP mainly consists of four optimal schemes, and respectively deals with four resource waste scenes of the job scheduling process. In the first scene, based on the job training method, an optimal scheme is designed to shield the interference of heterogeneous resources on job scheduling. In the second scene, we give two definition and develop another optimal scheme to dynamically adjust task number of multiple virtual machines on the same physical host. Through task adjustment, the high-capacity virtual machines deal more tasks than low-capacity ones. In addition, we build the overlapping execution model to overlap map, shuffle and reduce phases. In the third scene, considering the difference of map tasks and push tasks on resource usage needs, an overlapping scheme is formed to optimal the execution of push and map phase. In the last scene, a cross-jobs optimal scheme is proposed, which overlap execution of current job and next job. To avoid the bandwidth confliction between two jobs, a monitor method is used for reasonably resources allocation. Extensive experiments show that our algorithm consumes less job execution time and performs better than other three algorithms.